• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Memory management (streaming large fetches)
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Memory management (streaming large fetches)


  • Subject: Re: Memory management (streaming large fetches)
  • From: Ben Trumbull <email@hidden>
  • Date: Tue, 16 Mar 2004 17:00:34 -0800

The ideal way to do this is a little complicated.

If you override EOAdaptorChannel.fetchRow in a subclass (a bit easier to subclass JDBCChannel which is itself a subclass of EOAdaptorChannel), you can invoke super, process the row, and then return null to throw away that row.

Something like:
public NSMutableDictionary fetchRow() {
NSMutableDictionary row = super.fetchRow();
if(rowHandler != null){
rowHandler.processRow( row, this );
}
if (shouldAddRowsToResultArray ){
return row;
} else {
return null;
}
}


where rowHandler & shouldAddRowsToResultArray are private fields on your own subclass. When using your subclass, just set rowHandler to null and shouldAddRowsToResultArray to true to get the standard EOF behavior.

You may need to create your own (trivial) subclass of the JDBCAdaptor to hook in a custom channel subclass.

Other approaches will still cause EOF to hold the entire result set in memory until the end of the fetch operation, hence your problem with peak memory usage.

Another nice bit of this approach, is it works for any kind of fetch. You don't have to generate your own SQL query. If you throw away the results for a basic objectsWithFetchSpecification, EOF will simply think the query returned 0 rows.

The developers I know who have done this saw a peak memory usage of 10% the basic implementation (i.e. an order of magnitude improvement).

Of course, your mileage may vary :)

-Ben

At 19:09 +0100 3/16/04, Richard Bruch wrote:
Hi all,

a Java question to the smartest Java community in the world.

WO allows to quickly build HTML interface to a database so it seems reasonable
to use it to execute flexible data exports(XML or plain text for example).
However with Java(unlike C) it is not quite trivial if the tables(queries) are
large. I tried to export a subset of a large table using raw sql
EOAdaptorChannel methods. Even in that rather inexpensive case the memory
consumption after just 300 000 rows reaches 1 GB which is hardly acceptable.
Is there a means to deallocate memory(in this case the dictionaries returned
by EOAdaptorChannel.fetchRow()) in some predictable way or to force the
virtual machine to reclaim garbage ?

Or should i rather forget doing such things in Java ?

Thanks

Richard
_______________________________________________
webobjects-dev mailing list | email@hidden
Help/Unsubscribe/Archives: http://www.lists.apple.com/mailman/listinfo/webobjects-dev
Do not post admin requests to the list. They will be ignored.


References: 
 >Memory management (From: "Richard Bruch" <email@hidden>)

  • Prev by Date: Re: WebObjects 5.2.3 is now available
  • Next by Date: 5.2.3 Boolean Bugs?????
  • Previous by thread: Re: Memory management
  • Next by thread: RE: Memory management
  • Index(es):
    • Date
    • Thread