Re: Temporarily deaf instances
Re: Temporarily deaf instances
- Subject: Re: Temporarily deaf instances
- From: Gary Teter <email@hidden>
- Date: Wed, 26 Mar 2008 13:56:10 -0700
n Mar 26, 2008, at 1:39 PM, Mike Schrag wrote:
Where are those threads blocked? Most will be blocked waiting for
incoming web requests, i assume. Are there are threads blocked
anywhere other than chilling in their WorkThread wait state?
Can you hit the direct connect port during this time?
ms
Yes, many worker threads are normally blocked waiting for incoming
requests (on a normal, undeaf instance), but normally you'll see at
least one or more threads that are actually doing something, e.g.:
Ordinary worker thread waiting:
Thread t@4867: (state = BLOCKED)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=0,
line=382 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=50,
line=450 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame)
- com.webobjects.appserver._private.WOWorkerThread.run() @bci=26,
line=238 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
Ordinary thread doing something:
Thread t@4611: (state = IN_NATIVE)
- java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
byte[], int, int, int) @bci=0 (Compiled frame; information may be
imprecise)
- java.net.SocketInputStream.read(byte[], int, int) @bci=84,
line=129 (Compiled frame)
- org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill()
@bci=31, line=50 (Interpreted frame)
Deaf instance blocking at random point:
Thread t@37891: (state = BLOCKED)
- java.util.regex.Pattern.matcher(java.lang.CharSequence) @bci=25,
line=879 (Compiled frame)
- com.paizo.util.StringUtilities.stringWithoutHTMLEntities
(java.lang.String) @bci=423, line=2206 (Compiled frame)
- com.paizo.util.StringUtilities.htmlStringToStandardString
(java.lang.String) @bci=34, line=730 (Compiled frame)
Deaf instance blocking at random spot:
Thread t@30467: (state = BLOCKED)
- com.webobjects.eocontrol.EOGenericRecord._keyGetBindingForKey
(java.lang.String) @bci=10, line=171 (Compiled frame; information may
be imprecise)
- com.webobjects.eocontrol.EOCustomObject.valueForKey
(java.lang.String) @bci=8, line=1558 (Compiled frame)
- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
java.lang.Object[]) @bci=6, line=25 (Compiled frame)
- com.webobjects.eocontrol.EOCustomObject.valueForKey
(java.lang.String) @bci=14, line=1559 (Compiled frame)
- com.webobjects.foundation.NSKeyValueCoding$Utility.valueForKey
(java.lang.Object, java.lang.String) @bci=26, line=498 (Compiled frame)
- com.webobjects.foundation.NSKeyValueCodingAdditions
$DefaultImplementation.valueForKeyPath(java.lang.Object,
java.lang.String) @bci=19, line=212 (Compiled frame)
- com.webobjects.eocontrol.EOKeyValueQualifier.evaluateWithObject
(java.lang.Object) @bci=5, line=131 (Compiled frame)
- com.webobjects.eocontrol.EOQualifier.filteredArrayWithQualifier
(com.webobjects.foundation.NSArray,
com.webobjects.eocontrol.EOQualifier) @bci=55, line=635 (Compiled frame)
O
On Mar 26, 2008, at 4:37 PM, Gary Teter wrote:
Now I'm completely baffled. I've switched memcached clients, and
now I'm still getting temporarily deaf instances, but when I point
jstack at one, it now says that ALL threads are "state = BLOCKED".
After a couple of minutes the instance comes back.
Anyone have any ideas what could possibly be going on?
On Mar 25, 2008, at 4:18 PM, Chuck Hill wrote:
On Mar 25, 2008, at 4:05 PM, Gary Teter wrote:
jstack for the win! Not sure how I forgot about that tool, but
it was able to get me good info.
Chalk one up for jstack!
I switched to ActiveMQ, which is a billion times better than
OpenJMS, but we're still having temporarily deaf instances.
jstack reported that Every. Single. Thread. was "state =
BLOCKED", except one:
Thread t@18179: (state = IN_NATIVE)
- sun.nio.ch.KQueueArrayWrapper.kevent0(int, long, int, long)
@bci=0 (Interpreted frame)
- sun.nio.ch.KQueueArrayWrapper.poll(long) @bci=12, line=118
(Interpreted frame)
- sun.nio.ch.KQueueSelectorImpl.doSelect(long) @bci=46, line=69
(Interpreted frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
(Interpreted frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80
(Compiled frame)
- net.spy.memcached.MemcachedClient.run() @bci=11, line=1054
(Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot
(Frame.java:214)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP
(X86Frame.java:355)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender
(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:50)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
So it looks like I'm off to download a different memcached client.
--
WireHose: Smart metadata and personalization for WebObjects.
http://wirehose.com/
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden