Re: How to manage long running tasks
Re: How to manage long running tasks
- Subject: Re: How to manage long running tasks
- From: email@hidden
- Date: Tue, 6 Dec 2005 15:33:03 +0100
Chiming in with an all different solution....
We spin off long running task to separate processes.
These processes exists for task as various loading a file or generating
a report that takes 4 hours to compute.
We call our system of batch processes the workflow
system. A task is most often materialized as an entry in the database with
an attached state (say some-task.pending). Some processes also monitor
directories and work off files.
A workflow process is made of two threads: the producer
and the consumer. The consumer is made of a chain of worker objects taking
a workflow from one state to the next. We have some 30 of these processes.
A business task may actually be spread over several processes which guide
the workflow through intermediate states. It is thus possible to have several
concurrent worker processes running for the sub-tasks that tend to take
longer.
This architecture is coupled with the Workflow Manager.
The Workflow Manager is a WebObjects application managed by the WOTask.
For each process defined in the database it instantiates a copy of the
producer. Every 30 seconds or so it checks a producer to see if it matches
any record. If it does and if additional criteria (e.g. maximum number
of processes) are met, it spawn a worker process. Unless it is told by
the Workflow Manager to exit, the worker process will live as long as it
has work to do. Thus worker processes exist, use memory and CPU time, only
when there is actual work to do. Besides that only the Workflow Manager
lives permanently.
The Workflow Manager and the processes communicate
using direct actions. The Workflow Manager may also be controlled through
direct actions called from scripts. We for one allow it to use up more
resources at night than during office hours.
Hoping to have inspired some
Pierre
**********************************************************************
This email and any files transmitted with it are intended solely for
the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the sender
of this message. (email@hidden)
This email message has been checked for the presence of computer
viruses; however this protection does not ensure this message is
virus free.
Banque centrale du Luxembourg; Tel ++352-4774-1; http://www.bcl.lu
**********************************************************************
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden