Re: Temporarily deaf instances
Re: Temporarily deaf instances
- Subject: Re: Temporarily deaf instances
- From: Mike Schrag <email@hidden>
- Date: Wed, 26 Mar 2008 17:20:21 -0400
You can try turning on -verbose:gc ... It dumps HUGE amounts of
output, but you might beable to see a bunch of output dumping (or a
large GC size change) right around this pause time.
The format of the output is [GC sizebeforegc->sizeaftergc(memory
available), time it took to happen]
ms
On Mar 26, 2008, at 5:11 PM, Gary Teter wrote:
On Mar 26, 2008, at 2:00 PM, Mike Schrag wrote:
I wonder if jstack has to pause the VM to get a thread dump or
something? Maybe these are red herrings? If you jstack them when
they're NOT in this stack, do they say BLOCKED?
In a non-borked instance, most worker threads are blocked, but a few
are doing something. Those are usually IN_NATIVE.
Also, when it's in this state and you do TWO jstacks a little bit
apart in time, are there changes or is it like the VM is frozen in
time?
No changes according to diff.
ms
On Mar 26, 2008, at 4:56 PM, Gary Teter wrote:
n Mar 26, 2008, at 1:39 PM, Mike Schrag wrote:
Where are those threads blocked? Most will be blocked waiting
for incoming web requests, i assume. Are there are threads
blocked anywhere other than chilling in their WorkThread wait
state?
Can you hit the direct connect port during this time?
ms
Yes, many worker threads are normally blocked waiting for incoming
requests (on a normal, undeaf instance), but normally you'll see
at least one or more threads that are actually doing something,
e.g.:
Ordinary worker thread waiting:
Thread t@4867: (state = BLOCKED)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=0,
line=382 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=50,
line=450 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted
frame)
- com.webobjects.appserver._private.WOWorkerThread.run() @bci=26,
line=238 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
Ordinary thread doing something:
Thread t@4611: (state = IN_NATIVE)
- java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
byte[], int, int, int) @bci=0 (Compiled frame; information may be
imprecise)
- java.net.SocketInputStream.read(byte[], int, int) @bci=84,
line=129 (Compiled frame)
- org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill()
@bci=31, line=50 (Interpreted frame)
Deaf instance blocking at random point:
Thread t@37891: (state = BLOCKED)
- java.util.regex.Pattern.matcher(java.lang.CharSequence) @bci=25,
line=879 (Compiled frame)
-
com
.paizo
.util.StringUtilities.stringWithoutHTMLEntities(java.lang.String)
@bci=423, line=2206 (Compiled frame)
-
com
.paizo
.util.StringUtilities.htmlStringToStandardString(java.lang.String)
@bci=34, line=730 (Compiled frame)
Deaf instance blocking at random spot:
Thread t@30467: (state = BLOCKED)
-
com
.webobjects
.eocontrol.EOGenericRecord._keyGetBindingForKey(java.lang.String)
@bci=10, line=171 (Compiled frame; information may be imprecise)
-
com
.webobjects.eocontrol.EOCustomObject.valueForKey(java.lang.String)
@bci=8, line=1558 (Compiled frame)
-
sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
java.lang.Object[]) @bci=6, line=25 (Compiled frame)
-
com
.webobjects.eocontrol.EOCustomObject.valueForKey(java.lang.String)
@bci=14, line=1559 (Compiled frame)
- com.webobjects.foundation.NSKeyValueCoding
$Utility.valueForKey(java.lang.Object, java.lang.String) @bci=26,
line=498 (Compiled frame)
- com.webobjects.foundation.NSKeyValueCodingAdditions
$DefaultImplementation.valueForKeyPath(java.lang.Object,
java.lang.String) @bci=19, line=212 (Compiled frame)
-
com
.webobjects
.eocontrol
.EOKeyValueQualifier.evaluateWithObject(java.lang.Object) @bci=5,
line=131 (Compiled frame)
-
com
.webobjects
.eocontrol
.EOQualifier
.filteredArrayWithQualifier(com.webobjects.foundation.NSArray,
com.webobjects.eocontrol.EOQualifier) @bci=55, line=635 (Compiled
frame)
O
On Mar 26, 2008, at 4:37 PM, Gary Teter wrote:
Now I'm completely baffled. I've switched memcached clients, and
now I'm still getting temporarily deaf instances, but when I
point jstack at one, it now says that ALL threads are "state =
BLOCKED". After a couple of minutes the instance comes back.
Anyone have any ideas what could possibly be going on?
On Mar 25, 2008, at 4:18 PM, Chuck Hill wrote:
On Mar 25, 2008, at 4:05 PM, Gary Teter wrote:
jstack for the win! Not sure how I forgot about that tool, but
it was able to get me good info.
Chalk one up for jstack!
I switched to ActiveMQ, which is a billion times better than
OpenJMS, but we're still having temporarily deaf instances.
jstack reported that Every. Single. Thread. was "state =
BLOCKED", except one:
Thread t@18179: (state = IN_NATIVE)
- sun.nio.ch.KQueueArrayWrapper.kevent0(int, long, int, long)
@bci=0 (Interpreted frame)
- sun.nio.ch.KQueueArrayWrapper.poll(long) @bci=12, line=118
(Interpreted frame)
- sun.nio.ch.KQueueSelectorImpl.doSelect(long) @bci=46,
line=69 (Interpreted frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37,
line=69 (Interpreted frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80
(Compiled frame)
- net.spy.memcached.MemcachedClient.run() @bci=11, line=1054
(Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at
sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:214)
at
sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:
355)
at
sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:
134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:50)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
So it looks like I'm off to download a different memcached
client.
--
WireHose: Smart metadata and personalization for WebObjects.
http://wirehose.com/
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
--
WireHose: Smart metadata and personalization for WebObjects.
http://wirehose.com/
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden