Re: Memory management (streaming large fetches)
Re: Memory management (streaming large fetches)
- Subject: Re: Memory management (streaming large fetches)
- From: Ben Trumbull <email@hidden>
- Date: Wed, 17 Mar 2004 14:33:07 -0800
[Apologies if you received this twice.]
The ideal way to do this is a little complicated.
If you override EOAdaptorChannel.fetchRow in a subclass (a bit easier
to subclass JDBCChannel which is itself a subclass of
EOAdaptorChannel), you can invoke super, process the row, and then
return null to throw away that row.
Something like:
public NSMutableDictionary fetchRow() {
NSMutableDictionary row = super.fetchRow();
if(rowHandler != null){
rowHandler.processRow( row, this );
}
if (shouldAddRowsToResultArray ){
return row;
} else {
return null;
}
}
where rowHandler & shouldAddRowsToResultArray are private fields on
your own subclass. When using your subclass, just set rowHandler to
null and shouldAddRowsToResultArray to true to get the standard EOF
behavior.
You may need to create your own (trivial) subclass of the JDBCAdaptor
to hook in a custom channel subclass.
Other approaches will still cause EOF to hold the entire result set
in memory until the end of the fetch operation, hence your problem
with peak memory usage.
Another nice bit of this approach, is it works for any kind of fetch.
You don't have to generate your own SQL query. If you throw away the
results for a basic objectsWithFetchSpecification, EOF will simply
think the query returned 0 rows.
The developers I know who have done this saw a peak memory usage of
10% the basic implementation (i.e. an order of magnitude improvement).
Of course, your mileage may vary :)
-Ben
At 19:09 +0100 3/16/04, Richard Bruch wrote:
Hi all,
a Java question to the smartest Java community in the world.
WO allows to quickly build HTML interface to a database so it seems reasonable
to use it to execute flexible data exports(XML or plain text for example).
However with Java(unlike C) it is not quite trivial if the tables(queries) are
large. I tried to export a subset of a large table using raw sql
EOAdaptorChannel methods. Even in that rather inexpensive case the memory
consumption after just 300 000 rows reaches 1 GB which is hardly acceptable.
Is there a means to deallocate memory(in this case the dictionaries returned
by EOAdaptorChannel.fetchRow()) in some predictable way or to force the
virtual machine to reclaim garbage ?
Or should i rather forget doing such things in Java ?
Thanks
Richard
_______________________________________________
webobjects-dev mailing list | email@hidden
Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/webobjects-dev
Do not post admin requests to the list. They will be ignored.