Re: Search results in an out of memory exception
Re: Search results in an out of memory exception
- Subject: Re: Search results in an out of memory exception
- From: Jean-François Veillette <email@hidden>
- Date: Mon, 25 Oct 2004 16:25:18 -0400
This is because WODisplayGroup, will trigger the fetch of all eos even though it's displaying only 30 of them (in batch of 30 for example). Depending on the qualifier, you could end up fetching all the table in memory ... not a good thing for a large growing table. An eo take spaces in memory, in addition to the minimal spaces, it keep a db snapshot ( down in the eof layer ), eventually it will take some spaces in the undo manager. Spaces will also last longuer, as it won't be freed until the ec is freed (which for a default ec is long !).
Here a couple of other strategies, you can mix any of them with the one you have already to get a perfect solution :
- fetch raw rows, not eo(s), transform to eo latter, only once you need eos
- before doing a full fetch, do a select count(*) with the same where clause, to ge the number of row it's going to fetch, evaluate if it's a too big fetch, adjust your strategies
- look at WOnder ( or wocode / wodev ), i think there is a batch object somewhere that don't need all your eos in memory, with this object it will only fetch the eos displayed, the other batch object, will be fetch only if you need them.
- jfv
Le 04-10-25, à 13:17, David Holt a écrit :
I have just upped my test database of documents from 1500 records to 17,000. I have a WOComponent with a WODisplayGroup and the appropriate qualifier fields for searching. If I qualify the data source by putting a value in one of the search fields I get a list of documents as expected. If I submit the form without information in any of the qualifier fields (this used to return the whole data set divided into paged results), I get the following exception from the application after a minute or so of waiting:
Error:
com.webobjects.foundation.NSForwardException [java.lang.OutOfMemoryError] null
It is a MySQL database, WO 5.2.3, OS X Server 10.2
I have a blob field that holds the text content of the documents (for searching) as well as a URL field pointing to the original document on the file system. One of the qualifier fields is used to search the content field.
Three strategies I can think of to fix the problem are :
1. Increase system memory (not a good long term solution as the documents will grow over time)
2. Put the blob field in a separate table so it is not loaded with the WODisplayGroup (not sure if I can still do searches on that field if I do that)
3. Get the content out of the database and use a combination of PDFbox and Lucene to provide the content searching separate from my database.
What would you suggest is the best strategy? Or have I misidentified the problem ?
Thanks,
David
--
It's like driving a car at night. You never see further than your headlights, but you can make the whole trip that way.
E. L. Doctorow
from Sunbeams: http://www.thesunmagazine.org _______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden