RE: Threaded / offline processor (EOF Memory Management)
RE: Threaded / offline processor (EOF Memory Management)
- Subject: RE: Threaded / offline processor (EOF Memory Management)
- From: Andrew Lindesay <email@hidden>
- Date: Tue, 26 Apr 2005 09:29:35 +1200
Hello David;
Plan two - Direct actions
- uses the Direct Action session().defaultEditingContext()
- because the request had no sessionId, it would generate a new
session for each request.
If you do this, try make an EC (accessible via a static method) for use
in the direct action rather than using the one from a session as it
will then mean you don't need to create zillions of sessions that won't
be used. Don't forget to lock...unlock that EC using a try { ....}
finally { ... } around your processing.
Both processes were doing the same jobs, a simple HTML template using
EOF to fill with data, then FOP into PDF. Both use the same data from
the database over and over and over ...
I would try to;
1) create a fake WOContext
2) lock the D.A. EC
3) create the PDF data with the component using the D.A. EC
4) take off the PDF data into an NSData
5) unlock the EC
6) process the PDF data as need be
You may find that having the EC unlocked at step 5 and then some
ancillary processing happening in step 6 will allow the EC to get GC'ed
-- see below for further background.
Plan One - Thread Queues
I have achieved good batch-run performance for one of my customers by
doing a producer-consumer thing between instances and using JMS as the
queue transport mechanism. The consumers are on separate threads in
each instance. The only downside is that there is quite a bit of
rigging around consumer/producer management for shutdowns etc...
however if you're clever about your architecture around that, there is
considerable opportunity for reuse.
Unfortunately in these circumstances trying to understand the EOF
memory use patterns is not an easy game. Some of my processes involve
lots of EOF processing followed by some other I/O, CPU or JDBC
processing. I suspect this has let me off the hook in terms of memory
consumption as the second-phase processing has occured after the
editing context has been unlocked. Another reason why I might be
having a good run with this is that I intentionally try to keep
long-held data for the process in raw-rows. This means that such data
can be used outside of an unlocked editing context.
Recently however one of these things has been one-thread,EOF-only so it
goes....
ec.lock...process...ec.unlock
ec.lock...process...ec.unlock
ec.lock...process...ec.unlock
etc...
...and yes it has been observed that the memory usage will just keen on
ascending until an "out of memory" is reached. The only way in which I
could find to avoid this ghastly eventuality is to force a
garbage-collect after the "unlock" occurs.
I am fully aware that this is not a nice thing to do within the java
environment as it blocks all threads in the system for some time, but I
could find no other solution and it defintely works. I expect that
what is happening is that the editing context has the snapshots locked
so often that there simply is no chance for them to get GC'ed before
they get locked again. By forcing a GC I ensure that the
behind-the-scenes snapshots are lost from the storage (weak hash maps?)
in the editing context. I make the assumption here that no other EOF
lock is in place at this moment, but on average this should be suitable
to release enough memory to "carry on". I could imagine this would
also cause a degradation in performance as more faulting would need to
occur as well as the cost of making an explicit GC, but it's probably
still better to be able to use EOF and logic on existing EO's.
Has anybody else found another more elegant solution to this problem?
cheers.
___
Andrew Lindesay
www.lindesay.co.nz
+64-21-47-0929
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden