Re: Temporarily deaf instances
Re: Temporarily deaf instances
- Subject: Re: Temporarily deaf instances
- From: Chuck Hill <email@hidden>
- Date: Wed, 26 Mar 2008 14:11:38 -0700
That would explain why it comes back after a while.
On Mar 26, 2008, at 2:02 PM, Mike Schrag wrote:
A huge amount of garbage collection running, possibly while swapping
to disk might cause this ... How are you on memory consumption?
Have you allocated the VM a large amount of memory that might cause
it to swap to disk? You should see IO spike and memory be very
high. What is your VM -Xmx setting?
ms
On Mar 26, 2008, at 4:59 PM, Chuck Hill wrote:
That looks like the whole JVM locks up. I have _no_ idea what
might cause that!
Chuck
On Mar 26, 2008, at 1:56 PM, Gary Teter wrote:
n Mar 26, 2008, at 1:39 PM, Mike Schrag wrote:
Where are those threads blocked? Most will be blocked waiting
for incoming web requests, i assume. Are there are threads
blocked anywhere other than chilling in their WorkThread wait
state?
Can you hit the direct connect port during this time?
ms
Yes, many worker threads are normally blocked waiting for incoming
requests (on a normal, undeaf instance), but normally you'll see
at least one or more threads that are actually doing something,
e.g.:
Ordinary worker thread waiting:
Thread t@4867: (state = BLOCKED)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=0,
line=382 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=50,
line=450 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted
frame)
- com.webobjects.appserver._private.WOWorkerThread.run() @bci=26,
line=238 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
Ordinary thread doing something:
Thread t@4611: (state = IN_NATIVE)
- java.net.SocketInputStream.socketRead0(java.io.FileDescriptor,
byte[], int, int, int) @bci=0 (Compiled frame; information may be
imprecise)
- java.net.SocketInputStream.read(byte[], int, int) @bci=84,
line=129 (Compiled frame)
- org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill()
@bci=31, line=50 (Interpreted frame)
Deaf instance blocking at random point:
Thread t@37891: (state = BLOCKED)
- java.util.regex.Pattern.matcher(java.lang.CharSequence) @bci=25,
line=879 (Compiled frame)
-
com
.paizo
.util.StringUtilities.stringWithoutHTMLEntities(java.lang.String)
@bci=423, line=2206 (Compiled frame)
-
com
.paizo
.util.StringUtilities.htmlStringToStandardString(java.lang.String)
@bci=34, line=730 (Compiled frame)
Deaf instance blocking at random spot:
Thread t@30467: (state = BLOCKED)
-
com
.webobjects
.eocontrol.EOGenericRecord._keyGetBindingForKey(java.lang.String)
@bci=10, line=171 (Compiled frame; information may be imprecise)
-
com
.webobjects.eocontrol.EOCustomObject.valueForKey(java.lang.String)
@bci=8, line=1558 (Compiled frame)
-
sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object,
java.lang.Object[]) @bci=6, line=25 (Compiled frame)
-
com
.webobjects.eocontrol.EOCustomObject.valueForKey(java.lang.String)
@bci=14, line=1559 (Compiled frame)
- com.webobjects.foundation.NSKeyValueCoding
$Utility.valueForKey(java.lang.Object, java.lang.String) @bci=26,
line=498 (Compiled frame)
- com.webobjects.foundation.NSKeyValueCodingAdditions
$DefaultImplementation.valueForKeyPath(java.lang.Object,
java.lang.String) @bci=19, line=212 (Compiled frame)
-
com
.webobjects
.eocontrol
.EOKeyValueQualifier.evaluateWithObject(java.lang.Object) @bci=5,
line=131 (Compiled frame)
-
com
.webobjects
.eocontrol
.EOQualifier
.filteredArrayWithQualifier(com.webobjects.foundation.NSArray,
com.webobjects.eocontrol.EOQualifier) @bci=55, line=635 (Compiled
frame)
O
On Mar 26, 2008, at 4:37 PM, Gary Teter wrote:
Now I'm completely baffled. I've switched memcached clients, and
now I'm still getting temporarily deaf instances, but when I
point jstack at one, it now says that ALL threads are "state =
BLOCKED". After a couple of minutes the instance comes back.
Anyone have any ideas what could possibly be going on?
On Mar 25, 2008, at 4:18 PM, Chuck Hill wrote:
On Mar 25, 2008, at 4:05 PM, Gary Teter wrote:
jstack for the win! Not sure how I forgot about that tool, but
it was able to get me good info.
Chalk one up for jstack!
I switched to ActiveMQ, which is a billion times better than
OpenJMS, but we're still having temporarily deaf instances.
jstack reported that Every. Single. Thread. was "state =
BLOCKED", except one:
Thread t@18179: (state = IN_NATIVE)
- sun.nio.ch.KQueueArrayWrapper.kevent0(int, long, int, long)
@bci=0 (Interpreted frame)
- sun.nio.ch.KQueueArrayWrapper.poll(long) @bci=12, line=118
(Interpreted frame)
- sun.nio.ch.KQueueSelectorImpl.doSelect(long) @bci=46,
line=69 (Interpreted frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37,
line=69 (Interpreted frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80
(Compiled frame)
- net.spy.memcached.MemcachedClient.run() @bci=11, line=1054
(Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at
sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:214)
at
sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:
355)
at
sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:50)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
So it looks like I'm off to download a different memcached
client.
--
WireHose: Smart metadata and personalization for WebObjects.
http://wirehose.com/
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
--
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
--
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden