Re: Uploading a large file ties my application
Re: Uploading a large file ties my application
- Subject: Re: Uploading a large file ties my application
- From: Guido Neitzer <email@hidden>
- Date: Thu, 13 Nov 2008 14:24:13 -0700
On 13.11.2008, at 13:32, Ricardo J. Parada wrote:
Just wondering what is the best practice for uploading a file
without tying up the application? Each row in the file is inserted
to the database. So lots of inserts.
I would imagine someone has done already and there is a best
practice for this. Ideally, while the upload is in progress it
should not block the application from processing other requests...
and once the file has the data, or as the file is being processed
maybe with its own EOF-stack (connection to the database), I would
like to display a status that shows progress.
First, enable concurrent request handling in your app with:
-DWOAllowsConcurrentRequestHandling=true
to not block concurrent requests when one thread is working on an
upload or so. But set some useful numbers for the application settings
in JavaMonitor. For example I use:
Minimum Adaptor threads: 2
Maximum Adaptor threads: 8
Listen Queue Size: 4
The default values are way too high and if your app comes under some
very heavy load it will never recover when it grows the number of
worker threads too high and the listen queue piles up to 128 or so.
The perfect values for your app might be different from what I have
set, but it's a good idea to keep that low.
A better option is to use a different protocol for uploading large
files like scp or ftp which will not affect your apps. My app checks
specific upload folder in regular intervals (every minute) whether
there are files to process.
Next, imports. What I do there is:
- user uploads a big file either in the app or with ftp
- file is written to disk
A daemon process checks whether there are new files in an upload
folder, moves them to a "to be processed" folder and creates a
database entry for the file:
- an EO (ImportFile) is created and saved to the db that stores the
file path
- an Task object is created, the ImportFile attached and saved to the db
Most of that stuff can easily be done outside WO land, with cron, some
simple CLI scripts and so on. Now back to WO:
- task daemons check for task objects that are not "in progress"
- one daemon grabs a task, sets it to be "in progress", saves with
optimistic locking on the status column, the updateTime column and the
daemonName column, to prevent a successful saving when another daemon
was faster
- it starts processing the import
- when done it marks the task object as done and moves the file to a
location where it gets deleted a couple days later from a different
daemon, the reference in the ImportFile EO is updated and later marked
invalid
This is a little bit more complex system, I implemented it because I
needed to have several of those daemons processing files in parallel
and they are all pretty busy on specific times of the day ...
cug
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden