Re: Lots of EOs slow down the performance
Re: Lots of EOs slow down the performance
- Subject: Re: Lots of EOs slow down the performance
- From: Guido Neitzer <email@hidden>
- Date: Thu, 6 Nov 2008 10:17:01 -0700
On 06.11.2008, at 02:05, Yung-Luen Lan wrote:
First thing I tried is AjaxLongResponse. I thought it's acceptable to
display a progress bar and let users wait. Surprisingly, I still got a
"no instance error" after 150,000 eo created.
Sure, you probably got an out of memory exception if you don't recycle
your editing contexts and do some intelligent things for creating the
output.
Investigation more, actually, it's slow to do ec.saveChanges() with
150,000 eo inserted.
What do you expect? It has to create 150k insert statements. One for
each object, then copy these to the JDBC driver, that one executes
them one by one on the database ... and so on. It's just a plain
inefficient way of creating rows in the table.
I got broken pipe. I got heap memory error. I
have no idea why it break AjaxLongResponse. It's in another thread,
isn't it?
Again: you are running out of memory. Read the error message.
First of all: EOF is not build for bulk operations. If you want to do
something like that, you need to find other ways. What I prefer to do
for something like that:
- use ERXFetchSpecificationBatchIterator and iterate over the objects
in small batches of 100 or 200 rows
- create the CVS file on disk with an output stream that doesn't keep
the whole thing in memory
- deliver the file when the operation is done
Alternately, give the apps (and Eclipse) more memory - but that solves
it only to a certain point and is definitely not a preferable solution.
Just curious how do you deal with editingContext.saveChanges() with
lots of EO inserted? Um, I don't think 150,000 should be a big deal in
21 century.
Think about what is going on in that case. You have a 150000 objects
in Java land, you might have relationships, you create string
representations for each and every import statement, maybe more than
one string per statements - remember String is immutable and the GC
always comes too late -, you pass that to the JDBC adaptor as one
transaction so that one keeps it's own copy of the statements (not
sure, but likely if you expect the worst case), and so on. If each of
your objects is around 1k of size, you 150000k or around 146MB and
that is just to keep the objects around. What did you say how much
memory do you give your Java apps?
You should do some basic calculation and estimation before you
complain about something ... Just because you are using the tool
wrong, it's not bad or old. You still need to know what you're doing.
Especially with high level tools.
If you really need to do this in Java / EOF, recycle your editing
contexts every now and then. Better - use raw database operations. We
do imports of millions of objects each day and we have no problems ...
cug
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden