Threaded / offline processor (EOF Memory Management)
Threaded / offline processor (EOF Memory Management)
- Subject: Threaded / offline processor (EOF Memory Management)
- From: David Aspinall <email@hidden>
- Date: Mon, 25 Apr 2005 12:23:56 -0400
I have the need for a Cron style offline processor to run arbitrary
jobs. Some of the jobs will be simple (invoke a database stored
procedure), others will be more complex (synchronize account
information from 3rd party financial institutions using a XML web
service), and some will be in the middle (generate daily/weekly/monthly
reports)
We want the end users to be able to schedule and configure the 'jobs'
themselves, and the jobs must process even if they miss their
'run-time' because of system maintenance or error. So the basic job
definition and schedule is database driven. On the processor side I
have taken several approaches trying to find a workable one:
Plan One - Thread Queues
- one thread check the database schedule and enqueues jobs in a thread
safe queue.
- two or more threads wait for jobs in the queue and process as needed.
- each job allocates its own EOEditingContext with a new
EOObjectCoordinator (so each job should have a separate EOF stack)
- each job locks the EC, processes, then commits/rollsback/aborts,
then invalidates and 'disposes' of itself.
Plan two - Direct actions
- job requests come in as WS/direct action requests
- each request follows the standard action request logic (with
multithreaded request handling OFF)
- uses the Direct Action session().defaultEditingContext()
- because the request had no sessionId, it would generate a new
session for each request.
Plan Three - FULL JVM subprocesses
- each job runs as a separate process (not thread)
- OMG i feel sooooo heavy.
Plan Nine - drop EOF and go JDBC/JavaBeans/OpenSymphony Quartz
- ouch, there goes my business logic
I have built and tested Plans One and Two. Plan One goes like all get
out, chewing up through jobs non-stop. Unfortunately it will only go
about 3 hours before running out of memory and failing. And it fails
badly depending where the Out Of Memory Exception takes place. If it
is caught and discarded (especially in the MSSQL server JDBC adaptor) I
can get database deadlock, thread deadlock .... bad bad!
Plan 2 ran consistently with a steady 20 - 30Mb JVM footprint for over
3 hours. So I bumped up the request cycle (using cron and curl), and
then it started to go a little crazy also. I think it lived close to
10 hours (I had to go home and get married the next day, so I didn't
see the crash)
Both processes were doing the same jobs, a simple HTML template using
EOF to fill with data, then FOP into PDF. Both use the same data from
the database over and over and over ...
Does anyone have any experience in this area? Any suggestions for
getting better memory management out of EOF? I need this process to
last consistently and not fail badly.
Looking for help
David
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden