Re: Search results in an out of memory exception
Re: Search results in an out of memory exception
- Subject: Re: Search results in an out of memory exception
- From: Chuck Hill <email@hidden>
- Date: Mon, 25 Oct 2004 16:09:09 -0700
On Oct 25, 2004, at 3:37 PM, David Holt wrote:
Thank you,
I have done some more thinking about this and I have realized the
problem is that the WODisplayGroup is trying to display all 17,000
records.
That is probably more of a symptom than the problem.
I actually have to catch when the database is going to return too
many records. No-one in their right mind is going to try and browse
through that many records without some sort of search strategy.
I've seen many clients insist that they need to do this. :-) But
you're right, no one in their right mind...
I think that Jean-François nails it when he says to do a count first
and modify the strategy based on the size of the found set. I assume
that I'll have to write my own fetches to be able to do this?
That all depends on what your strategy is.
The other question I have is whether Lucene can actually return
results to a WORepetition that I can link to my Document Detail page,
or if it returns just a standard "google" like page that I only link
directly to the original .pdf? Thanks again, you guys are awesome.
Yep, you just back a list of document IDs. Your code then has to
construct a fault for each document (or the ones you want), and shove
them into an array. EOF will fetch the data when you touch (fire) each
fault, or you can fault in the whole array. You can, of course, get
even more fancy.
Chuck
David
--
I don't want you to follow me or anyone else. I would not lead you to
the promised land if I could, because if I could lead you in, somebody
else would lead you out.
–Eugene V. Debs
from Sunbeams: http://www.thesunmagazine.org
On 25 Oct 2004, at 1:25 PM, Jean-François Veillette wrote:
This is because WODisplayGroup, will trigger the fetch of all eos
even though it's displaying only 30 of them (in batch of 30 for
example). Depending on the qualifier, you could end up fetching all
the table in memory ... not a good thing for a large growing table.
An eo take spaces in memory, in addition to the minimal spaces, it
keep a db snapshot ( down in the eof layer ), eventually it will take
some spaces in the undo manager. Spaces will also last longuer, as
it won't be freed until the ec is freed (which for a default ec is
long !).
Here a couple of other strategies, you can mix any of them with the
one you have already to get a perfect solution :
- fetch raw rows, not eo(s), transform to eo latter, only once you
need eos
- before doing a full fetch, do a select count(*) with the same where
clause, to ge the number of row it's going to fetch, evaluate if it's
a too big fetch, adjust your strategies
- look at WOnder ( or wocode / wodev ), i think there is a batch
object somewhere that don't need all your eos in memory, with this
object it will only fetch the eos displayed, the other batch object,
will be fetch only if you need them.
- jfv
Le 04-10-25, à 13:17, David Holt a écrit :
I have just upped my test database of documents from 1500 records to
17,000. I have a WOComponent with a WODisplayGroup and the
appropriate qualifier fields for searching. If I qualify the data
source by putting a value in one of the search fields I get a list
of documents as expected. If I submit the form without information
in any of the qualifier fields (this used to return the whole data
set divided into paged results), I get the following exception from
the application after a minute or so of waiting:
Error:
com.webobjects.foundation.NSForwardException
[java.lang.OutOfMemoryError] null
It is a MySQL database, WO 5.2.3, OS X Server 10.2
I have a blob field that holds the text content of the documents
(for searching) as well as a URL field pointing to the original
document on the file system. One of the qualifier fields is used to
search the content field.
Three strategies I can think of to fix the problem are :
1. Increase system memory (not a good long term solution as the
documents will grow over time)
2. Put the blob field in a separate table so it is not loaded with
the WODisplayGroup (not sure if I can still do searches on that
field if I do that)
3. Get the content out of the database and use a combination of
PDFbox and Lucene to provide the content searching separate from my
database.
What would you suggest is the best strategy? Or have I misidentified
the problem ?
Thanks,
David
--
It's like driving a car at night. You never see further than your
headlights, but you can make the whole trip that way.
E. L. Doctorow
from Sunbeams: http://www.thesunmagazine.org
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
email@hidden
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
village.net
This email sent to email@hidden
--
Practical WebObjects - a book for intermediate WebObjects developers
who want to increase their overall knowledge of WebObjects, or those
who are trying to solve specific application development problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden