Re: Lots of EOs slow down the performance
Re: Lots of EOs slow down the performance
- Subject: Re: Lots of EOs slow down the performance
- From: "Yung-Luen Lan" <email@hidden>
- Date: Fri, 7 Nov 2008 05:33:15 +0800
On Fri, Nov 7, 2008 at 1:17 AM, Guido Neitzer <email@hidden> wrote:
> On 06.11.2008, at 02:05, Yung-Luen Lan wrote:
> Sure, you probably got an out of memory exception if you don't recycle your
> editing contexts and do some intelligent things for creating the output.
You're right. I don't know if it's proper to handle ECs in my way.
Normally I set a variable to hold EC in each components, sometimes I
create EOs, insert them into a new EC and track the EC with
object.editingContext(). Do you have any trick or pattern to recycle
EC? Call System.gc()?
>> Investigation more, actually, it's slow to do ec.saveChanges() with
>> 150,000 eo inserted.
> What do you expect? It has to create 150k insert statements. One for each
> object, then copy these to the JDBC driver, that one executes them one by
> one on the database ... and so on. It's just a plain inefficient way of
> creating rows in the table.
I'm not sure about this. Maybe group those SQL statements into one
transaction could help?
(Or is that already been done with EOF?)
> Again: you are running out of memory. Read the error message.
>
> First of all: EOF is not build for bulk operations. If you want to do
> something like that, you need to find other ways. What I prefer to do for
> something like that:
>
> - use ERXFetchSpecificationBatchIterator and iterate over the objects in
> small batches of 100 or 200 rows
>
> - create the CVS file on disk with an output stream that doesn't keep the
> whole thing in memory
>
> - deliver the file when the operation is done
Yeah, those are actually what I do taking your suggestions--except the
first one. I'm still learning what is a fetch specification. Thanks
again.
> Alternately, give the apps (and Eclipse) more memory - but that solves it
> only to a certain point and is definitely not a preferable solution.
>> Just curious how do you deal with editingContext.saveChanges() with
>> lots of EO inserted? Um, I don't think 150,000 should be a big deal in
>> 21 century.
> Think about what is going on in that case. You have a 150000 objects in Java
> land, you might have relationships, you create string representations for
> each and every import statement, maybe more than one string per statements -
> remember String is immutable and the GC always comes too late -, you pass
> that to the JDBC adaptor as one transaction so that one keeps it's own copy
> of the statements (not sure, but likely if you expect the worst case), and
> so on. If each of your objects is around 1k of size, you 150000k or around
> 146MB and that is just to keep the objects around. What did you say how much
> memory do you give your Java apps?
I totally agree your point. Let me break this into two parts--space and time.
Memory: it appears that EC isn't suited for insert a lot EO into
database because the memory footstep of EC work like this: hold
objects in memory, discard or save them to db at once. Other ORM tool
like activerecord or python.db seems don't have the concept of EC;
they just invoke save() on each objects. In this case, activerecord or
python.db are not likely devour all the memory because each object
could be collected after they're saved. I'm not saying that which one
is better than which. EC is not for doing bulk insert, just like you
said. I just want to know if there is way to save each object
seperately. Finally I put a line into loop and it works: if (i % 1000
== 0) ec.saveChangs();
Performance: I did some benchmark on my database. 150,000 insertion on
the same table:
Raw SQL, transaction: 23s
Raw SQL, no transaction: 154s
EC, saveChanges every 1000 EO inserted: 273s
Comparing raw SQL without transaction, EC method is not bad at all.
(only 1.7x slower) I don't care about to reduce the wait time from
five minutes to half. Totally acceptable. :-)
> You should do some basic calculation and estimation before you complain
> about something ... Just because you are using the tool wrong, it's not bad
> or old. You still need to know what you're doing. Especially with high level
> tools.
Ah, if my previous post offense people, I apologize. What I really
mean is obsolutely not "why webobject is bad and old", but "This
should be done easily in 21 century. I must do something wrong. What's
the correct way?"
> If you really need to do this in Java / EOF, recycle your editing contexts
> every now and then. Better - use raw database operations. We do imports of
> millions of objects each day and we have no problems ...
>
> cug
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden