Re: Search results in an out of memory exception
Re: Search results in an out of memory exception
- Subject: Re: Search results in an out of memory exception
- From: David Holt <email@hidden>
- Date: Mon, 25 Oct 2004 15:37:58 -0700
Thank you,
I have done some more thinking about this and I have realized the problem is that the WODisplayGroup is trying to display all 17,000 records. I actually have to catch when the database is going to return too many records. No-one in their right mind is going to try and browse through that many records without some sort of search strategy. I think that Jean-François nails it when he says to do a count first and modify the strategy based on the size of the found set. I assume that I'll have to write my own fetches to be able to do this? The other question I have is whether Lucene can actually return results to a WORepetition that I can link to my Document Detail page, or if it returns just a standard "google" like page that I only link directly to the original .pdf? Thanks again, you guys are awesome.
David
<x-tad-smaller>--
I don't want you to follow me or anyone else. I would not lead you to the promised land if I could, because if I could lead you in, somebody else would lead you out.
–Eugene V. Debs
from Sunbeams: </x-tad-smaller><x-tad-smaller>http://www.thesunmagazine.org</x-tad-smaller><x-tad-smaller> </x-tad-smaller>
On 25 Oct 2004, at 1:25 PM, Jean-François Veillette wrote:
This is because WODisplayGroup, will trigger the fetch of all eos even though it's displaying only 30 of them (in batch of 30 for example). Depending on the qualifier, you could end up fetching all the table in memory ... not a good thing for a large growing table. An eo take spaces in memory, in addition to the minimal spaces, it keep a db snapshot ( down in the eof layer ), eventually it will take some spaces in the undo manager. Spaces will also last longuer, as it won't be freed until the ec is freed (which for a default ec is long !).
Here a couple of other strategies, you can mix any of them with the one you have already to get a perfect solution :
- fetch raw rows, not eo(s), transform to eo latter, only once you need eos
- before doing a full fetch, do a select count(*) with the same where clause, to ge the number of row it's going to fetch, evaluate if it's a too big fetch, adjust your strategies
- look at WOnder ( or wocode / wodev ), i think there is a batch object somewhere that don't need all your eos in memory, with this object it will only fetch the eos displayed, the other batch object, will be fetch only if you need them.
- jfv
Le 04-10-25, à 13:17, David Holt a écrit :
I have just upped my test database of documents from 1500 records to 17,000. I have a WOComponent with a WODisplayGroup and the appropriate qualifier fields for searching. If I qualify the data source by putting a value in one of the search fields I get a list of documents as expected. If I submit the form without information in any of the qualifier fields (this used to return the whole data set divided into paged results), I get the following exception from the application after a minute or so of waiting:
Error:
com.webobjects.foundation.NSForwardException [java.lang.OutOfMemoryError] null
It is a MySQL database, WO 5.2.3, OS X Server 10.2
I have a blob field that holds the text content of the documents (for searching) as well as a URL field pointing to the original document on the file system. One of the qualifier fields is used to search the content field.
Three strategies I can think of to fix the problem are :
1. Increase system memory (not a good long term solution as the documents will grow over time)
2. Put the blob field in a separate table so it is not loaded with the WODisplayGroup (not sure if I can still do searches on that field if I do that)
3. Get the content out of the database and use a combination of PDFbox and Lucene to provide the content searching separate from my database.
What would you suggest is the best strategy? Or have I misidentified the problem ?
Thanks,
David
--
It's like driving a car at night. You never see further than your headlights, but you can make the whole trip that way.
E. L. Doctorow
from Sunbeams: http://www.thesunmagazine.org _______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden