Re: Temporarily deaf instances
Re: Temporarily deaf instances
- Subject: Re: Temporarily deaf instances
- From: Mike Schrag <email@hidden>
- Date: Wed, 26 Mar 2008 16:39:58 -0400
Where are those threads blocked? Most will be blocked waiting for
incoming web requests, i assume. Are there are threads blocked
anywhere other than chilling in their WorkThread wait state?
Can you hit the direct connect port during this time?
ms
On Mar 26, 2008, at 4:37 PM, Gary Teter wrote:
Now I'm completely baffled. I've switched memcached clients, and now
I'm still getting temporarily deaf instances, but when I point
jstack at one, it now says that ALL threads are "state = BLOCKED".
After a couple of minutes the instance comes back.
Anyone have any ideas what could possibly be going on?
On Mar 25, 2008, at 4:18 PM, Chuck Hill wrote:
On Mar 25, 2008, at 4:05 PM, Gary Teter wrote:
jstack for the win! Not sure how I forgot about that tool, but it
was able to get me good info.
Chalk one up for jstack!
I switched to ActiveMQ, which is a billion times better than
OpenJMS, but we're still having temporarily deaf instances.
jstack reported that Every. Single. Thread. was "state = BLOCKED",
except one:
Thread t@18179: (state = IN_NATIVE)
- sun.nio.ch.KQueueArrayWrapper.kevent0(int, long, int, long)
@bci=0 (Interpreted frame)
- sun.nio.ch.KQueueArrayWrapper.poll(long) @bci=12, line=118
(Interpreted frame)
- sun.nio.ch.KQueueSelectorImpl.doSelect(long) @bci=46, line=69
(Interpreted frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
(Interpreted frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled
frame)
- net.spy.memcached.MemcachedClient.run() @bci=11, line=1054
(Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at
sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:214)
at
sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP(X86Frame.java:355)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:
218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:50)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
So it looks like I'm off to download a different memcached client.
On Mar 24, 2008, at 12:32 PM, Mike Schrag wrote:
That's what everyone says, but JGroups made this particular
problem worse. :(
Hmm .. I've been running the JGroups version for almost two years
in production and with some pretty substantial load tests
(500-1000 insane-clicky-concurrent-making-changing users). I've
never run it on top of JMS, and admittedly, you lose persistent
message delivery with the non-JMS one, but for us this wasn't a
problem.
But still, I don't think this is a deadlock because you come
back, and you don't come back from deadlocks. JGroups does love
to create threads, but they don't have many live concurrently,
generally (just lots created over time). Without being able to
get a thread dump, it will be pretty hard to diagnose this. If
you're running on 1.5 (I think you must be because I think
JGroups is 1.5 only?) you might try running a jstack on your app
when this happens, though I would think it would have less
success than a kill -QUIT would ... Worth a shot, though.
--
WireHose: Smart metadata and personalization for WebObjects.
http://wirehose.com/
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
--
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
--
WireHose: Smart metadata and personalization for WebObjects.
http://wirehose.com/
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden