Re: Lots of EOs slow down the performance
Re: Lots of EOs slow down the performance
- Subject: Re: Lots of EOs slow down the performance
- From: Chuck Hill <email@hidden>
- Date: Thu, 6 Nov 2008 09:30:56 -0800
On Nov 6, 2008, at 9:17 AM, Guido Neitzer wrote:
On 06.11.2008, at 02:05, Yung-Luen Lan wrote:
First thing I tried is AjaxLongResponse. I thought it's acceptable to
display a progress bar and let users wait. Surprisingly, I still
got a
"no instance error" after 150,000 eo created.
Sure, you probably got an out of memory exception if you don't
recycle your editing contexts and do some intelligent things for
creating the output.
Investigation more, actually, it's slow to do ec.saveChanges() with
150,000 eo inserted.
What do you expect? It has to create 150k insert statements.
And another 150K select statements to get the PK for the new rows!
Chuck
One for each object, then copy these to the JDBC driver, that one
executes them one by one on the database ... and so on. It's just a
plain inefficient way of creating rows in the table.
I got broken pipe. I got heap memory error. I
have no idea why it break AjaxLongResponse. It's in another thread,
isn't it?
Again: you are running out of memory. Read the error message.
First of all: EOF is not build for bulk operations. If you want to
do something like that, you need to find other ways. What I prefer
to do for something like that:
- use ERXFetchSpecificationBatchIterator and iterate over the
objects in small batches of 100 or 200 rows
- create the CVS file on disk with an output stream that doesn't
keep the whole thing in memory
- deliver the file when the operation is done
Alternately, give the apps (and Eclipse) more memory - but that
solves it only to a certain point and is definitely not a preferable
solution.
Just curious how do you deal with editingContext.saveChanges() with
lots of EO inserted? Um, I don't think 150,000 should be a big deal
in
21 century.
Think about what is going on in that case. You have a 150000 objects
in Java land, you might have relationships, you create string
representations for each and every import statement, maybe more than
one string per statements - remember String is immutable and the GC
always comes too late -, you pass that to the JDBC adaptor as one
transaction so that one keeps it's own copy of the statements (not
sure, but likely if you expect the worst case), and so on. If each
of your objects is around 1k of size, you 150000k or around 146MB
and that is just to keep the objects around. What did you say how
much memory do you give your Java apps?
You should do some basic calculation and estimation before you
complain about something ... Just because you are using the tool
wrong, it's not bad or old. You still need to know what you're
doing. Especially with high level tools.
If you really need to do this in Java / EOF, recycle your editing
contexts every now and then. Better - use raw database operations.
We do imports of millions of objects each day and we have no
problems ...
cug
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden