Re: Temporarily deaf instances
Re: Temporarily deaf instances
- Subject: Re: Temporarily deaf instances
- From: Gary Teter <email@hidden>
- Date: Tue, 25 Mar 2008 16:05:03 -0700
jstack for the win! Not sure how I forgot about that tool, but it was
able to get me good info.
I switched to ActiveMQ, which is a billion times better than OpenJMS,
but we're still having temporarily deaf instances. jstack reported
that Every. Single. Thread. was "state = BLOCKED", except one:
Thread t@18179: (state = IN_NATIVE)
- sun.nio.ch.KQueueArrayWrapper.kevent0(int, long, int, long)
@bci=0 (Interpreted frame)
- sun.nio.ch.KQueueArrayWrapper.poll(long) @bci=12, line=118
(Interpreted frame)
- sun.nio.ch.KQueueSelectorImpl.doSelect(long) @bci=46, line=69
(Interpreted frame)
- sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
(Interpreted frame)
- sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled
frame)
- net.spy.memcached.MemcachedClient.run() @bci=11, line=1054
(Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot
(Frame.java:214)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP
(X86Frame.java:355)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:
218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:50)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
So it looks like I'm off to download a different memcached client.
On Mar 24, 2008, at 12:32 PM, Mike Schrag wrote:
That's what everyone says, but JGroups made this particular
problem worse. :(
Hmm .. I've been running the JGroups version for almost two years
in production and with some pretty substantial load tests (500-1000
insane-clicky-concurrent-making-changing users). I've never run it
on top of JMS, and admittedly, you lose persistent message delivery
with the non-JMS one, but for us this wasn't a problem.
But still, I don't think this is a deadlock because you come back,
and you don't come back from deadlocks. JGroups does love to
create threads, but they don't have many live concurrently,
generally (just lots created over time). Without being able to get
a thread dump, it will be pretty hard to diagnose this. If you're
running on 1.5 (I think you must be because I think JGroups is 1.5
only?) you might try running a jstack on your app when this
happens, though I would think it would have less success than a
kill -QUIT would ... Worth a shot, though.
--
WireHose: Smart metadata and personalization for WebObjects.
http://wirehose.com/
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden