Re: Invalidating objects asynchronously
Re: Invalidating objects asynchronously
- Subject: Re: Invalidating objects asynchronously
- From: Dan Grec <email@hidden>
- Date: Mon, 8 Sep 2008 17:47:00 -0600
On 3-Sep-08, at 4:16 PM, Mike Schrag wrote:
We're having an interesting problem in our apps, and I wanted to
see if anyone else has solved it.
Yes, with an asterisk. As Travis said, use Project Wonder and the
ERXRemoteSynchronizer with JGroups support ... The JMS change
notification framework is known to have problems under load. I have
tested the remote synchronizer under VERY heavy load tests (500-1000
insanely clicking updating multiple instances) and verified for
accuracy of results (accuracy defined as "final state in all
instances matches the database"). If JMS is invalidating objects,
it's going to be screwed -- you CANNOT invalidate and expect a valid
graph state ... We take a performance hit, but in the case where an
EO is faulted into the snapshot cache and then modified in a remote
instance, we force a refresh from the DB. There is really almost no
other way to end up in a safe state (I tried LOTS of approaches -- I
rewrote this entire process in 3 different ways until I found all
the crazy cases that work). We also properly update inverse
relationships (which every other change notification system that
I've seen does incorrectly). I'm not saying you can't end up in a
potentially wonky state, because this ultimately is impossible to
get 100% right, but I've tested all the other popular ones and I can
tell you for a 100% fact that they get it wrong. You WILL get
deadlocks and you WILL get incorrect graph state. Wonder's is the
only one that has was able to get through my butt-wuppin' load test
and survive, and I've been using this in a multiple instance
application in production for almost 2 years without any graph state
exceptions.
As far as sharding your EO's, I definitely agree ... If your data
lends itself to sharding, then this is far more ideal (and
dependable ... if you get it right). However, there are some cases
where you have globally collaborative data that does not lend itself
to sharding, or even if it does, you may want to have a shard on
multiple instances, which puts you back where you started. There
are also often applications that require users to cross shards,
which means you have to have a system for dealing with redirecting
the user to another instance mid-flight. All completely possible to
do, but none of these solutions are trivial (but then we wouldn't
have jobs if they were :) ).
ms
Mike,
Thanks very much for the reply. As always, your knowledge and help is
invaluable.
We're getting deeper and deeper down the rabbit hole here...
deployment date is looming, and we've spent almost two weeks on this
one bug that is really really hurting.
I have a few very specific questions before we move forward, any help
is greatly appreciated.
1. We're thinking that the only way to fix this is to start using
Project Wonder and the ERXRemoteSynchronizer for change notifications.
I'm still a little concerned that it won't help our specific case.
(here is the scary scenario)
We get an NSArray of EO's from a relationship, and are enumerating
over them.
A change notification comes in relating to one of the EO's in the
array (it's actually been deleted by another user on another instance,
which happens all the time)
When the object in question is re-fetched from the database, it turns
out to be deleted(!) and is then null, and is then very bad for us.
Will ERXRemoteSynchronizer handle this is some nice way, or will the
object still resolve to null in the middle of the enumeration? (if
it's still null, we need to think of some other solution...)
(so if it's still going to get nulled in the middle of the Enumeration
- what the heck can we do?)
2. We ditched JGroups in favor of JMS as our message transport so we
can use the built-in JMS providers in our J2EE deploys. (which is much
nicer)
We're thinking that we can write an interface for the constructor of
ERXRemoteSynchronizer so that it can use a JGroups or JMS provider
transparently.
In your reply above you said that JMS will not work, ever.
Can you please expand on this? I was under the impression that JMS is
used and relied upon heavily in the J2EE world.
Will JMS + ERXRemoteSynchronizer be OK? (which I would think is the
case, as JMS just delivers messages, the Wonder framework decides
what to do about the message).
Thanks once again,
-Dan Grec
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden