• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Starting second instance=app not available
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Starting second instance=app not available

  • Subject: Re: Starting second instance=app not available
  • From: Jeffrey Schmitz <email@hidden>
  • Date: Tue, 04 Mar 2014 02:30:14 -0600

Did a reboot of machine to "clear things out" and not can't restart any instances!!!

The app starts with no errors, it's just that javamonitor doesn't see it started.

Here's what womonitor.log looks like:

2014-03-04 03:28:41,506 DEBUG 16.15 MB/17.78 MB [WorkerThread13] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - @@@@@ monitorRequestAction requestDict: {queryWotaskd = "INSTANCE"; }

2014-03-04 03:28:41,506 DEBUG 16.15 MB/17.78 MB [WorkerThread13] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - @@@@@ monitorRequestAction returning response to Monitor
2014-03-04 03:28:41,506 DEBUG 16.15 MB/17.78 MB [WorkerThread13] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - @@@@@ monitorRequestAction responseDict: {queryWotaskdResponse = {instanceResponse = ({port = 2003; runningState = "DEAD"; statistics = {}; id = 1; nextShutdown = "-"; applicationName = "netBrackets"; refusingNewSessions = false; host = "netbrackets1.netbrackets.com"; deaths = ("03/04/2014 03:27:23 America/New_York"); }, {port = 2004; runningState = "DEAD"; statistics = {}; id = 2; nextShutdown = "-"; applicationName = "netBrackets"; refusingNewSessions = false; host = "netbrackets1.netbrackets.com"; deaths = (); }, {port = 2001; runningState = "DEAD"; statistics = {}; id = 3; nextShutdown = "-"; applicationName = "netBrackets"; refusingNewSessions = false; host = "netbrackets1.netbrackets.com"; deaths = (); }); }; }

2014-03-04 03:28:41,511 DEBUG 11.53 MB/22.41 MB [WorkerThread3] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - ##### pageWithName(AppDetailPage) errors: ()
2014-03-04 03:28:41,513 DEBUG 11.57 MB/22.37 MB [WorkerThread3] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - message(): ()
2014-03-04 03:28:41,513 INFO  11.57 MB/22.37 MB [WorkerThread3] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:40) - Session.message: null
2014-03-04 03:28:41,514 DEBUG 11.61 MB/22.33 MB [WorkerThread3] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - message(): ()
2014-03-04 03:28:41,514 INFO  11.61 MB/22.33 MB [WorkerThread3] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:40) - Session.message: null
2014-03-04 03:28:45,528 DEBUG 16.23 MB/17.71 MB [WorkerThread2] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - @@@@@ Received Lifebeat: lifebeat netBrackets netbrackets1.netbrackets.com 2003
2014-03-04 03:28:53,235 DEBUG 16.23 MB/17.71 MB [main] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - _checkAutoRecover START
2014-03-04 03:28:53,238 DEBUG 16.23 MB/17.71 MB [WorkerThread0] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - @@@@@ Received Lifebeat: lifebeat JavaMonitor netbrackets1.netbrackets.com 56789
2014-03-04 03:28:53,238 DEBUG 16.23 MB/17.71 MB [main] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - _checkAutoRecover STOP
2014-03-04 03:29:15,531 DEBUG 16.27 MB/17.66 MB [WorkerThread2] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - @@@@@ Received Lifebeat: lifebeat netBrackets netbrackets1.netbrackets.com 2003
2014-03-04 03:29:23,274 DEBUG 16.27 MB/17.66 MB [main] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - _checkAutoRecover START
2014-03-04 03:29:23,275 DEBUG 16.27 MB/17.66 MB [main] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - _checkAutoRecover STOP
2014-03-04 03:29:23,277 DEBUG 16.32 MB/17.62 MB [WorkerThread0] logging.ERXNSLogLog4jBridge (ERXNSLogLog4jBridge.java:46) - @@@@@ Received Lifebeat: lifebeat JavaMonitor netbrackets1.netbrackets.com 56789


On Mar 3, 2014, at 9:03 PM, Chuck Hill <email@hidden> wrote:

Those are use normal idle WOWorkerThreads waiting for an incoming request.  You won’t get any useful information until the apps are not responding.



On 2014-03-03, 7:01 PM, "Jeffrey Schmitz" wrote:

On my other machine that's running 10.8.5 server with the "same" build and  jvm:

netbrandeds-mini1:~ netBrackets$ java -version
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)


I get better results.  Here's a snippet:


sudo jstack -F 22627
Password:
Attaching to process ID 22627, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.65-b04-462
Deadlock Detection:

No deadlocks found.

Thread t@26319: (state = BLOCKED)
 - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=0, line=437 (Interpreted frame)
 - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=468 (Interpreted frame)
 - java.net.ServerSocket.accept() @bci=48, line=436 (Interpreted frame)
 - com.webobjects.appserver._private.WOWorkerThread.run() @bci=26, line=210 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=695 (Interpreted frame)


Thread t@43727: (state = IN_NATIVE)
 - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame)
 - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=439 (Interpreted frame)
 - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=468 (Interpreted frame)
 - java.net.ServerSocket.accept() @bci=48, line=436 (Interpreted frame)
 - com.webobjects.appserver._private.WOWorkerThread.run() @bci=26, line=210 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=695 (Interpreted frame)


Thread t@23047: (state = BLOCKED)
 - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=0, line=437 (Interpreted frame)
 - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=468 (Interpreted frame)
 - java.net.ServerSocket.accept() @bci=48, line=436 (Interpreted frame)
 - com.webobjects.appserver._private.WOWorkerThread.run() @bci=26, line=210 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=695 (Interpreted frame)




On Mar 3, 2014, at 8:48 PM, Chuck Hill <email@hidden> wrote:

I don’t know what is going on with jstack.  Did you build with debugging info stripped out?  Or this could be a Mavericks issue.



On 2014-03-03, 6:37 PM, "Jeffrey Schmitz" wrote:

OS is Maverics Server.  JVM is 
netbrackets1:~ netbrackets$ java -version
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)


ps auxwww ... gives:

netbrackets1:logs netbrackets$ ps auxwww | grep netBrackets
_appserver      62709   0.1  3.9  6066292 329292   ??  S    12:28AM   2:20.24 /usr/bin/java -XX:NewSize=2m -Xmx64m -Xms32m -DWORootDirectory=/System -DWOLocalRootDirectory= -DWOUserDirectory=/Library/WebObjects/JavaApplications/wotaskd.woa -DWOEnvClassPath= -DWOApplicationClass=com.netbrackets.app.Application -DWOPlatform=MacOS -Dcom.webobjects.pid=62709 -Xms2048m -Xmx3072m -classpath WOBootstrap.jar com.webobjects._bootstrap.WOBootstrap -WOPort 2004 -WOCachingEnabled YES -WODebuggingEnabled NO -WOOutputPath /Library/WebObjects/Logs/netBrackets-2 -WOAutoOpenInBrowser NO -WOAutoOpenClientApplication NO -WOLifebeatInterval 30 -WOLifebeatEnabled YES -WOLifebeatDestinationPort 1085 -WOAdaptor WODefaultAdaptor -WOWorkerThreadCount 8 -WOListenQueueSize 8 -WOWorkerThreadCountMin 2 -WOWorkerThreadCountMax 16 -NSProjectSearchPath () -WOSessionTimeOut 3600 -WOApplicationName netBrackets -WOMonitorEnabled YES -WONoPause YES -d64 -WOAllowsConcurrentRequestHandling YES EOAdaptorDebugEnabled NO -WODirectConnectEnabled NO -WOFrameworksBaseURL /WebObjects/Frameworks
root            36486   0.0  0.0  2490648   3200   ??  S    Sun02PM   0:00.09 sshd: netBrackets [priv] 
root            15866   0.0  0.0  2498840   3264   ??  S    Sun06AM   0:00.19 sshd: netBrackets [priv] 
root              109   0.0  0.3  2500916  21752   ??  S     9Feb14  17:18.66 /Library/FrontBase/bin/FrontBase /Library/FrontBase/Databases/netBrackets.fb
netbrackets     17066   0.0  0.0  2432784    636 s000  S+    9:31PM   0:00.00 grep netBrackets


So, jstack on 62709 gives the below long dump, and now I'm in way over my head:



Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = IN_NATIVE)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)


Thread t@-1: (state = BLOCKED)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.debugger.macosx.MacOSXThread.getContext(MacOSXThread.java:87)
at sun.jvm.hotspot.runtime.macosx_amd64.MacOSXAMD64JavaThreadPDAccess.getCurrentFrameGuess(MacOSXAMD64JavaThreadPDAccess.java:92)
at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)
On Mar 3, 2014, at 8:29 PM, Chuck Hill <email@hidden> wrote:



On 2014-03-03, 6:20 PM, "Jeffrey Schmitz" wrote:

Thought I'd do a jstack on the instance that is out there, so I looked for its process ID with:

ps aux | grep netBrackets

Probably one is JavaMonitor or something.  Try 
ps auxwww | grep netBrackets


And 2 processes showed up with that name even though only one is showing in javamonitor?!?
root            36486   0.0  0.0  2490648   3200   ??  S    Sun02PM   0:00.09 sshd: netBrackets [priv] 
root            15866   0.0  0.0  2498840   3264   ??  S    Sun06AM   0:00.19 sshd: netBrackets [priv] 

I did a jstack on them and got the same on both:

netbrackets1:logs netbrackets$ sudo jstack -F 15866
Attaching to process ID 15866, please wait...
sun.jvm.hotspot.debugger.NoSuchSymbolException: Could not find symbol "gHotSpotVMTypes" in any of the known library names (-)
at sun.jvm.hotspot.HotSpotTypeDataBase.lookupInProcess(HotSpotTypeDataBase.java:398)
at sun.jvm.hotspot.HotSpotTypeDataBase.readVMTypes(HotSpotTypeDataBase.java:104)
at sun.jvm.hotspot.HotSpotTypeDataBase.<init>(HotSpotTypeDataBase.java:85)
at sun.jvm.hotspot.MacOSXTypeDataBase.<init>(MacOSXTypeDataBase.java:36)
at sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:578)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:499)
at sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:337)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.tools.jstack.JStack.runJStackTool(JStack.java:118)
at sun.tools.jstack.JStack.main(JStack.java:84)

What OS and JVM are you running on?

Try
sudo kill –QUIT <process ID>

If you have SpawnOfWotaskd.sh setup correctly, the dump should appear in the app log.


Chuck




On Mar 3, 2014, at 7:52 PM, Chuck Hill <email@hidden> wrote:



On 2014-03-03, 5:49 PM, "Jeffrey Schmitz" wrote:

Will have to wait until later tonight when things quiet down for the jstack command

I do have another server setup with the same load that serves a different site and that is functioning fine.

Something is different.  :-)


Would I see any stack traces if it's deadlock?  I'm not seeing anything in any of the log files.

Yes.



Isolation level is set to:
dbConnectURLGLOBAL=jdbc:FrontBase://localhost/netBrackets/isolation=read_committed/locking=optimistic

Looks correct.



Should I be using localhost or the IP or the domain name here?

Should not matter.


Chuck




Thanks!
Jeff

On Mar 3, 2014, at 7:25 PM, Chuck Hill <email@hidden> wrote:

Do a 
Sudo jstack –F <process ID>
On the instances when this happens.  My guess is that they are contending for a resource and neither is able to process requests.   Deadlock would be my guess, maybe the wrong isolation level on the DB connection? This then gets escalated to dead and mis-reported as No Instance Available.


Chuck


On 2014-03-03, 4:57 PM, "Jeffrey Schmitz" wrote:

Hello,
   Here's one I don't think I've seen.  When I start a second instance on my server (mac OS Maverick Server), I start getting app not avaliable errors.  Even if I try to get to the original instance directly through the url (append /1), I still get app not available.  If I then set the new instance to not accept new sessions, or kill it, then the original instance will start taking requests again.  Am hoping if nothing else a reboot tonight will fix this situation, but would be good to know what's really going on, and it worries that maybe the reboot won't fix it and I'll be hosed.

Note the new instances start fine, no errors are logged and everything looks good in JavaMonitor.  Also, this just started.  The server has been running for a couple weeks and I've been starting and stopping new instances all along (using Bounce) just fine.  Any ideas?

Thanks!
Jeff
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden


 

 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden
  • Follow-Ups:
    • Re: Starting second instance=app not available
      • From: Alexander Spohr <email@hidden>
References: 
 >Starting second instance=app not available (From: Jeffrey Schmitz <email@hidden>)
 >Re: Starting second instance=app not available (From: Chuck Hill <email@hidden>)
 >Re: Starting second instance=app not available (From: Jeffrey Schmitz <email@hidden>)
 >Re: Starting second instance=app not available (From: Chuck Hill <email@hidden>)
 >Re: Starting second instance=app not available (From: Jeffrey Schmitz <email@hidden>)
 >Re: Starting second instance=app not available (From: Chuck Hill <email@hidden>)
 >Re: Starting second instance=app not available (From: Jeffrey Schmitz <email@hidden>)
 >Re: Starting second instance=app not available (From: Chuck Hill <email@hidden>)
 >Re: Starting second instance=app not available (From: Jeffrey Schmitz <email@hidden>)
 >Re: Starting second instance=app not available (From: Chuck Hill <email@hidden>)

  • Prev by Date: Re: Starting second instance=app not available
  • Next by Date: Re: Unique Constraint Error
  • Previous by thread: Re: Starting second instance=app not available
  • Next by thread: Re: Starting second instance=app not available
  • Index(es):
    • Date
    • Thread