Re: Running a time consuming background process
Re: Running a time consuming background process
- Subject: Re: Running a time consuming background process
- From: Stephen Hoffman <email@hidden>
- Date: Sun, 27 Jan 2008 16:12:48 -0500
- Organization: HoffmanLabs LLC
I need to run a custom program that I expect to need several weeks of
(wall clock) time to complete.
How can I set up and run so that I can save state on any signal, and
having my program automatically resumed
on start up, with as little extra work as possibly when resuming.
My program is a pure batch job, by the way.
Why not checkpoint your progress and your run-time state on a periodic
basis?
This approach is simple, easier to verify, and catches a hard power-down
and most anything else.
In my experience, there are cases where catching errors can be too
clever by half. What happens if you catch an error that means you
can't, for instance, write?
How and what to checkpoint varies greatly, based on what you are doing.
Transactional processing and databases work well here for a subset of
the possible cases.
This is less efficient (proportional to the quantity of data and the
frequency of the checkpoints), but this approach is also easier to code,
test and maintain.
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Darwin-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden